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Mr.  Chairman  and  Members  of  the  Subcommittee: 


I  am  pleased  to  be  here  today  to  discuss  the  Department  of  Defense’s 
(DOD)  management  of  its  investment  in  the  Standard  Procurement  System 
or  SPS  program.  The  department  launched  this  program  a  little  more  than 
7  years  ago  with  the  laudable  goal  of  replacing  76  existing  procurement 
systems  with  a  single  departmentwide  system  to  more  effectively  support 
divergent  contracting  processes  and  procedures  across  its  component 
organizations.  Through  SPS,  the  department  expected  to  improve 
efficiency  and  effectiveness  in  how  it  awarded  and  managed  contracts, 
and,  at  that  time,  estimated  life-cycle  costs  to  be  approximately  $3  billion 
over  a  10-year  period. 

The  department’s  goals  for  SPS  are  reinforced  by  the  president’s  recent 
management  agenda,  which  emphasizes  investing  in  information 
technology  to  achieve  results.  The  agenda  also  noted  that  the  federal 
government  has  not  produced  measurable  gains  in  productivity 
commensurate  with  its  investment  in  information  technology, 1  which  is 
now  estimated  to  be  more  than  $50  billion  for  fiscal  year  2003.  The  agenda 
reiterates  that  program  performance  and  results  are  what  matters  most, 
and  that  actual  program  accomplishments,  as  well  as  needs,  should  be  the 
prerequisite  to  continued  funding.  This  emphasis  is  consistent  with 
information-technology  investment  management  provisions  of  federal  law 
and  guidance2  and  information-technology  management  practices  of 
leading  public-  and  private-sector  companies. 

For  the  SPS  program,  we  reported  in  July  2001  that  the  department  had 
not  met  these  investment  management  criteria.3  Specifically: 

•  The  department  had  not  economically  justified  its  investment  in  the 
program  because  its  latest  (January  2000)  analysis  of  costs  and 
benefits  was  not  credible.  Further,  this  flawed  analysis  showed  that 
the  system,  as  defined,  was  not  a  cost-beneficial  investment. 

•  It  had  not  effectively  addressed  the  inherent  risks  associated  with 
investing  in  a  program  as  large  and  lengthy  as  SPS  because  it  had  not 
divided  the  program  into  incremental  investment  decisions  that 
coincided  with  incremental  releases  of  system  capabilities. 


1  The  President’s  Management  Agenda:  Fiscal  Year 2002,  Executive  Office  of  the  President, 
Office  of  Management  and  Budget. 

2Clinger-Cohen  Act  of  1996,  Public  Law  104-106;  Office  of  Management  and  Budget  Circular 
A-130,  Management  of  Federal  Information  Resources  (November  30,  2000). 

3U.S.  General  Accounting  Office,  DOD  Systems  Modernization:  Continued  Investment  in  the 
Standard  Procurement  System  Has  Not  Been  Justified,  GAO-Ol-682  (Washington,  D.C.:  July 
31,  2001). 
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•  The  department  had  not  met  key  program  commitments  that  were 
used  to  justify  the  program.  For  example,  the  department  committed 
to  implementing  a  commercially  available  contract  management 
system;  however,  because  it  had  modified  so  much  of  the  foundational 
commercial  product,  SPS  evolved  into  a  customized  DOD  system. 

Also,  although  the  department  committed  to  fully  implementing  the 
system  by  March  31,  2000,  this  target  date  had  slipped  by  3  Vz  years  to 
September  30,  2003,  and  program  officials  have  recently  stated  that 
this  date  will  also  not  be  met. 

•  It  did  not  know  if  it  was  meeting  other  key  program  commitments.  For 
example,  the  department  had  not  measured  whether  promised  system 
benefits  were  being  realized,  and  the  information  that  was  available 
about  system  performance  showed  that  users  were  not  satisfied  with 
the  system.  Also,  because  DOD  was  not  accumulating  actual  program 
costs,  it  did  not  know  the  total  amount  spent  on  the  program  to  date, 
yet  life-cycle  cost  projections  had  grown  from  about  $3  billion  to  $3.7 
billion. 

Collectively,  this  meant  that  the  question  of  whether  further  investment  in 
SPS  was  justified  could  not  be  answered  with  any  certainty.  Accordingly, 
we  recommended  that  investment  in  future  releases  or  major 
enhancements  to  the  system  be  made  conditional  on  the  department  first 
demonstrating  that  the  system  was  producing  benefits  that  exceed  costs, 
and  that  future  investment  decisions  be  based  on  complete  and  reliable 
economic  justifications.  We  also  recommended  that  program  officials 
clarify  organizational  accountability  and  responsibility  for  the  program, 
determine  the  program’s  current  status,  and  identify  lessons  learned  from 
the  SPS  investment  management  experience. 

In  commenting  on  a  draft  of  our  report,  the  Deputy  Chief  Information 
Officer  (CIO)  generally  disagreed  with  our  recommendations,  noting  that 
they  would  delay  development  and  deployment  of  SPS.  Since  that  time, 
however,  the  department  has  either  initiated  or  stated  its  intention  to 
initiate  steps  that  are  consistent  with  our  recommendations.  It  has  also 
taken  steps  to  address  the  findings  of  several  department-sponsored 
studies  initiated  at  the  time  of  our  report.  For  example,  it  has  (1)  clarified 
organizational  accountability  and  responsibility  for  the  program, 

(2)  established  missing  controls  over  key  acquisition  processes  such  as 
requirements  management  and  testing,  and  (3)  begun  addressing  users’ 
concerns.  In  addition,  department  officials  have  stated  that  the  department 
will  prepare  an  economic  analysis  before  investing  beyond  already 
executed  contractual  commitments  and  that  it  will  conduct  a  productivity 
study  to  assess  the  extent  to  which  the  department  is  deriving  benefits 
from  SPS.  These  are  positive  steps  that  have  advanced  the  program 
beyond  where  it  was  at  the  time  of  our  report. 
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SPS:  A  Brief 
Description  and 
History 


Nevertheless,  much  remains  to  be  done  before  the  department  will  be  in  a 
position  to  make  an  informed,  data-driven  decision  about  whether  further 
investment  in  the  system  is  justified.  Namely,  although  program  officials 
have  stated  their  intentions  to  address  our  recommendations,  they  have 
not  yet  committed  to  specific  tasks  for  doing  so  nor  have  they  established 
milestone  dates  for  completing  these  tasks.  Further,  the  department  may 
expand  the  functionality  of  the  current  software  release  to  include 
requirements  previously  slated  for  later  releases,  which  could  compound 
existing  problems  and  increase  costs;  and,  although  intended  to  be  a 
standard  system  for  the  entire  department,  not  all  defense  components 
have  agreed  to  adopt  SPS. 


In  November  1994,  the  Office  of  the  Director  of  Defense  Procurement 
initiated  the  SPS  program  to  acquire  and  deploy  a  single  automated  system 
to  perform  all  contract-management-related  functions  for  all  DOD 
organizations.  At  that  time,  life-cycle  costs  were  estimated  to  be  about  $3 
billion  over  a  10-year  period. 

From  1994  to  1996,  the  department  defined  SPS  requirements  and  solicited 
commercially  available  vendor  products  for  satisfying  these  requirements. 
Subsequently,  in  April  1997,  the  department  awarded  a  contract  to 
American  Management  Systems  (AMS),  Incorporated,  to  (1)  use  AMS’s 
commercially  available  contract  management  system  as  the  foundation  for 
SPS,  (2)  modify  this  commercial  product  as  necessary  to  meet  DOD 
requirements,  and  (3)  perform  related  services.4  The  department  also 
directed  the  contractor  to  deliver  functionality  for  the  system  in  four 
incremental  releases.  The  department  later  increased  the  number  of 
releases  across  which  this  functionality  would  be  delivered  to  seven, 
reduced  the  size  of  the  increments,  and  allowed  certain  more  critical 
functionality  to  be  delivered  sooner  (see  table  1  for  proposed  SPS 
functionality  by  increment). 


“DOD  is  not  acquiring  the  source  code  for  SPS  and,  unless  an  expanded  license  is  obtained, 
is  required  to  obtain  sole-source  support  over  the  life  of  this  system  from  AMS. 
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Table  1:  Summary  of  SPS  Functionality  by  Increment 

Increment 

Software 

release 

(subreleases) 

Functionality 

1 

3.1 

Provide  base-level  contracting  capabilities  enabling 

DOD  procurement  personnel  to  prepare  simple 
contracts,  which  are  generally  fixed-price,  1-year 
contracts  that  will  not  be  modified. 

2 

3 

4 

5 

3.5 

4.0 

4.1  (a-e) 

4.2 

Provide  enhanced  base-level  contracting  functionality 
for  DOD  procurement  personnel,  such  as  reporting 
and  contract  administration  capabilities,  automatic 
edits,  security  features,  and  electronic  interfaces  for 
legacy  systems  being  replaced. 

6 

5.0 

Provide  more  complex  contracting  capabilities, 
enabling  DOD  procurement  personnel  to  purchase 
weapons  systems.  These  contracts  are  generally 
fewer  in  number,  but  are  more  complicated,  consisting 
of  numerous  provisions  and  contract  line-item 
numbers,  and  usually  undergo  extensive 
modifications. 

7 

5.1 

Provide  inventory  control  point  (ICP)  functionality  for 
ICPs,  which  are  responsible  for  the  support  and 
acquisition  of  spare  parts  and  supplies,  enabling 
workload  management  to  better  manage  inventories. 

Source:  DOD. 


Since  our  report  of  July  2001, 5  DOD  has  revised  its  plans.  According  to  the 
SPS  program  manager,  current  plans  no  longer  include  increments  6  and  7 
or  releases  5.0  and  5.1.  Instead,  release  4.2  (increment  5)  will  include  at 
least  three,  but  not  more  than  seven,  subreleases.  At  this  time,  only  the 
first  of  the  potentially  seven  4.2  subreleases  is  under  contract.  This 
subrelease  is  scheduled  for  delivery  in  April  2002,  with  deployment  to  the 
Army  and  the  Defense  Logistics  Agency  scheduled  for  June  2002.  Based  on 
the  original  delivery  date,  release  4.2  is  about  one  year  overdue. 

The  department  reports  that  it  has  yet  to  define  the  requirements  to  be 
included  within  the  remaining  4.2  subreleases,  and  has  not  executed  any 
contract  task  orders  for  these  subreleases.  According  to  SPS  officials,  they 
will  decide  later  this  year  whether  to  invest  in  these  additional  releases. 

As  of  December  2001,  the  department  reported  that  it  had  deployed  four 
SPS  releases  to  over  777  locations.*1  The  Director  of  Defense  Procurement 
(DDP)  has  responsibility  for  the  SPS  program,'  and  the  CIO  is  the 


‘GAO-0 1-682  (July  31,  2001). 

‘All  DOD  components  except  the  Air  Force  have  deployed  subrelease  4.1e;  the  Air  Force 
has  only  deployed  through  subrelease  4.1b.  The  Air  Force  is  scheduled  to  begin  deployment 
of  release  4.1e  in  March  2002. 

DDP  is  organizationally  located  within  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics. 
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milestone  decision  authority  for  SPS  because  the  program  is  classified  as  a 
major  Defense  acquisition.8 


Numerous  SPS 
Concerns  Have  Been 
Raised  by  Us  and 
Others 


Our  July  2001  report  detailed  program  problems  and  investment 
management  weaknesses.9  To  address  these  weaknesses,  we 
recommended,  among  other  things,  that  the  department  report  on  the 
lessons  to  be  learned  from  its  SPS  experience  for  the  benefit  of  future 
system  acquisitions.  Similarly,  other  reviews  of  the  program 
commissioned  by  the  department  in  the  wake  of  our  review  raised  similar 
concerns  and  identified  other  problems  and  management  weaknesses.  The 
findings  from  our  report  are  summarized  below  in  two  major  categories: 
lack  of  economic  justification  for  the  program  and  inability  to  meet 
program  commitments.  We  also  summarize  the  findings  of  the  other 
studies. 


The  Clinger-Cohen  Act  of  1996,  OMB  guidance,  DOD  policy,  and  practices 
of  leading  organizations  provide  an  effective  framework  for  managing 
information  technology  investments,  not  just  when  a  program  is  initiated, 
but  continuously  throughout  the  life  of  the  program.  Together,  they 
provide  for 

(1)  economically  justifying  proposed  projects  on  the  basis  of  reliable 
analyses  of  expected  life-cycle  costs,  benefits,  and  risks;  and 

(2)  using  these  analyses  throughout  a  project’s  life-cycle  as  the  basis  for 
investment  selection,  control,  and  evaluation  decisionmaking,  and 
doing  so  for  large  projects  (to  the  maximum  extent  practical)  by 
dividing  them  into  a  series  of  smaller,  incremental  subprojects  or 
releases  and  individually  justifying  investment  in  each  separate 
increment  on  the  basis  of  costs,  benefits,  and  risks. 

The  department  had  not  met  these  investment  management  tenets  for  SPS. 
First,  the  latest  economic  analysis  for  the  program — dated  January  2000 — 
was  not  based  on  reliable  estimates  because  most  of  the  cost  estimates  in 
the  2000  economic  analysis  were  estimates  carried  forward  from  the  April 
1997  analysis  (adjusted  for  inflation).  Only  the  cost  estimates  being  funded 
and  managed  by  the  SPS  program  office,  which  were  13  percent  of  the 
total  estimated  life-cycle  cost  in  the  analysis,  were  updated  in  2000  to 
reflect  more  current  contract  estimates  and  actual  expenditures/ 
obligations  for  fiscal  years  1995  through  1999.  Moreover,  the  military 
services,  which  share  funding  responsibility  with  the  SPS  program  office 
for  implementing  the  program,  questioned  the  reliability  of  these  cost 


DOD  Had  Not 
Economically  Justified 
Its  Investment  in  SPS 


sDOD  Regulation  5000. 2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition  Programs 
and  Major  Automated  Information  System  Acquisition  Programs,  specifies  mandatory 
policies  and  procedures  for  major  acquisitions.  The  policy  also  specifies  that  the  DOD  CIO 
is  the  milestone  decision  authority,  responsible  for  program  approval,  for  all  major 
automated  information  systems,  such  as  SPS. 

9GAO-0 1-682  (July  31,  2001). 
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estimates.  However,  this  uncertainty  was  not  reflected  in  the  economic 
analysis  using  any  type  of  sensitivity  analysis.10  A  sensitivity  analysis  would 
have  disclosed  for  decisionmakers  the  investment  risk  being  assumed  by 
relying  on  the  estimates  presented  in  the  economic  analysis. 

Moreover,  the  latest  economic  analysis  (January  2000)  was  outdated 
because  it  did  not  reflect  the  program’s  current  status  and  known 
problems  and  risks.  For  instance,  this  analysis  was  based  on  a  program 
scope  and  associated  costs  and  benefits  that  anticipated  four  software 
releases.  However,  as  mentioned  previously,  the  program  now  consists  of 
five  releases,  and  subreleases  within  releases,  in  order  to  accommodate 
changes  in  SPS  requirements.  Estimates  of  the  full  costs,  benefits,  and 
risks  relating  to  this  additional  release  and  its  subreleases  were  not  part  of 
the  2000  economic  analysis.  Also,  this  analysis  did  not  fully  recognize 
actual  and  expected  delays  in  meeting  SPS’s  full  operational  capability 
milestone,  which  had  been  slipped  by  3%  years  and  DOD  officials  say  that 
further  delays  are  currently  expected.  Such  delays  not  only  increase  the 
system  acquisition  costs  but  also  postpone,  and  thus  reduce,  accrual  of 
system  benefits.  Further,  several  DOD  components  are  now  questioning 
whether  they  will  even  deploy  the  software,  which  would  further  reduce 
SPS’s  cost  effectiveness  calculations  in  the  2000  economic  analysis. 

Second,  the  department  had  not  used  these  analyses  as  the  basis  for 
deciding  whether  to  continue  to  invest  in  the  program.  The  latest 
economic  analysis  showed  that  SPS  was  not  a  cost-beneficial  investment 
because  the  estimated  benefits  to  be  realized  did  not  exceed  estimated 
program  costs.  In  fact,  the  2000  analysis  showed  estimated  costs  of  $3.7 
billion  and  estimated  benefits  of  $1.4  billion,  which  was  a  recovery  of  only 
37  percent  of  costs.  According  to  the  former  SPS  program  manager,  this 
analysis  was  not  used  to  manage  the  program  and  there  was  no  DOD 
requirement  for  updating  an  economic  analysis  when  changes  to  the 
program  occurred. 

Third,  DOD  had  not  made  its  investment  decisions  incrementally  as 
required  by  the  Clinger-Cohen  Act  and  OMB  guidance.  That  is,  although 
the  department  is  planning  to  acquire  and  implement  SPS  as  a  series  of 
five  increments,  it  has  not  made  decisions  about  whether  to  invest  in  each 
release  on  the  basis  of  the  release’s  expected  return  on  investment,  as  well 
as  whether  prior  releases  were  actually  achieving  return-on-investment 
expectations.  In  fact,  for  the  four  increments  that  have  been  deployed,  the 
department  had  not  validated  whether  the  increments  were  providing 
promised  benefits  and  was  not  accounting  for  the  costs  associated  with 
each  increment  so  that  it  could  even  determine  actual  return  on 
investment. 


“That  is,  an  analysis  to  explicitly  present  the  return-on-investment  implications  associated 
with  using  estimates  whose  inherent  imprecision  could  produce  a  range  of  outcomes. 
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Instead,  the  department  had  treated  investment  in  this  program  as  one, 
monolithic  investment  decision,  justified  by  a  single,  “all-or-nothing” 
economic  analysis.  Our  work  has  shown  that  it  is  difficult  to  estimate,  with 
any  degree  of  accuracy,  cost  and  schedule  estimates  for  many  increments 
to  be  delivered  over  many  years  because  later  increments  are  not  well 
understood  or  defined.  Also,  these  estimates  are  subject  to  change  based 
on  actual  program  experiences  and  changing  requirements.  This  “all-or- 
nothing”  approach  to  investing  in  large  system  acquisitions,  like  SPS,  has 
repeatedly  proven  to  be  ineffective  across  the  federal  government, 
resulting  in  huge  sums  being  invested  in  systems  that  do  not  provide 
commensurate  benefits. 


DOD  Had  Not  Met  or 
Did  Not  Know  if  It  Had 
Met  SPS  Commitments 


Measuring  progress  against  program  commitments  is  closely  aligned  with 
economically  justifying  information-technology  investments,  and  is  equally 
important  to  ensuring  effective  investment  management.  The  Clinger- 
Cohen  Act,  OMB  guidance,  DOD  policy, 11  and  practices  of  leading 
organizations  provide  for  making  and  using  such  measurements  as  part  of 
informed  investment  decisionmaking. 

DOD  had  not  met  key  commitments  and  was  uncertain  whether  it  was 
meeting  other  commitments  because  it  was  not  measuring  them.  (See 
table  2  for  a  summary  of  the  department’s  progress  against  commitments.) 


uDOD  Interim  Regulation  5000.2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  and  Major  Automated  Information  System  Acquisition  Programs  (January  4, 
2001). 
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Table  2:  Progress  Against  SPS  Program  Commitments 

Key  commitments 

Commitment  met? 

Explanation(s) 

System  fully  operational  by  March  31 , 
2000 

No 

Problems  were  encountered  in  modifying  and  testing  the 
commercial  product  and  in  adequately  defining  requirements.  For 
example,  there  were  no  system  performance  requirements  in  the 
SPS  contract.aThe  tarqet  date  had  slipped  3-1/2  years. b 

Contracting  community’s  needs  met 

No 

Approximately  60  percent  of  the  user  population  recently  surveyed 
by  DOD’s  OIG  were  dissatisfied  with  the  system’s  functionality  and 
performance.0 

Acquire  a  commercially  available 
software  product 

No 

The  commercial  product  had  been  extensively  modified,  resulting 
in  a  DOD-unique  system. 

Other  commitments 

Replace  76  legacy  procurement 
systems  and  manual  processes,  thereby 
reducing  procurement  system 
operations  and  maintenance  costs 

? 

Only  2  legacy  systems  had  been  fully  retired  and  2  partially  retired, 
and  DOD  did  not  know  what,  if  any,  associated  cost  savings  had 
resulted.  Also,  DOD  now  plans  to  retire  only  14  legacy  systems  as 
a  result  of  SPS’s  implementation. 

Increase  user  productivity 

? 

DOD  is  unaware  of  the  extent  to  which  productivity  may  have 
increased  because  it  did  not  implement  needed  performance 
metrics. 

Standardize  policies,  processes,  and 
procedures 

? 

Each  military  service  had  or  was  planning  to  develop  its  own 
unique  program  documentation. 

Reduce  problem  disbursements 

? 

DOD  was  unable  to  provide  any  evidence  that  implementing  SPS 
had  reduced  problem  disbursements,  nor  had  it  included  this 
benefit  in  its  latest  economic  analysis. 

Life-cycle  costs  of  $3.7  billion  over  a  10- 
year  period 

? 

DOD  was  unaware  of  the  amount  spent  on  the  program  to  date 
because  cost  information  was  being  tracked  and  officially  reported 
only  for  the  SPS  program  office.  Costs  incurred  by  all  DOD 
component  organizations  were  not  accumulated  and  reported. d 

? — DOD  was  unaware  of  the  extent  to  which  the  commitment  had  been  met. 

“While  the  former  program  manager  attributed  the  delay  to  an  increase  in  requirements,  the  SPS  Joint  Requirements  Board  chairperson 
stated  that  no  additional  requirements  had  been  approved.  Rather,  the  board’s  chairperson  stated  that  the  original  requirements  had 
not  been  well-defined  and  clarification  was  needed  to  better  ensure  that  user  needs  would  be  met. 

bAccording  to  the  current  program  manager,  the  most  recent  target  date  of  September  30,  2003,  will  not  be  met.  In  addition,  another 
target  date  has  not  yet  been  established  for  completing  the  program. 

°A  user  satisfaction  manager  was  recently  designated  for  this  program. 

dBased  on  DOD  documents  we  obtained  during  our  current  review,  at  a  minimum,  $51 1 .6  million  had  been  spent  as  of  September  30, 
2001. 


To  partially  fill  the  void  in  knowing  progress  against  SPS  commitments, 
the  program  office  initiated  a  study  in  June  2000  to  validate  the  extent  to 
which  benefits  from  version  4.1  would  be  realized.  However,  the  study  was 
not  well  planned  and  executed,  and  while  some  useful  information  was 
obtained,  the  study  did  not  allow  DOD  to  validate  whether  expected 
benefits  were  actually  being  realized.  For  example, 

•  the  sample  selected  was  not  statistically  valid,  meaning  that  the  results 
were  not  projectable  to  the  population  as  a  whole, 

•  the  study  was  based  on  the  1997  economic  analysis  instead  of  the  more 
current  2000  economic  analysis,  despite  key  differences  between  the 
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two  analyses,  such  as  the  number  and  dollar  value  of  estimated 
benefits,  and 

•  the  information  gathered  did  not  map  to  the  22  benefit  types  listed  in 
the  1997  economic  analysis.  Instead,  the  study  collected  subjective 
judgments  (perceptions)  that  were  not  based  on  predefined 
performance  metrics  for  SPS  capabilities  and  impacts.  Thus,  the 
department  was  not  measuring  SPS  against  its  promised  benefits. 

The  former  program  manager  told  us  that  knowing  whether  SPS  was 
producing  value  and  meeting  commitments  was  not  the  program  office’s 
objective  because  there  was  no  departmental  requirement  to  do  so. 

Rather,  the  objective  was  simply  to  acquire  and  deploy  the  system. 
Similarly,  CIO  officials  told  us  that  the  department  was  not  validating 
whether  deployed  releases  of  SPS  were  producing  benefits  because  there 
was  no  DOD  requirement  to  do  so  and  no  metrics  had  been  defined  for 
such  validation.12  However,  the  Clinger-Cohen  Act  of  1996  and  OMB 
guidance13  emphasize  the  need  to  have  investment  management  processes 
and  information  to  help  ensure  that  information-technology  projects  are 
being  implemented  at  acceptable  costs  and  within  reasonable  and 
expected  time  frames  and  that  they  are  contributing  to  tangible, 
observable  improvements  in  mission  performance  (i.e.,  that  projects  are 
meeting  the  cost,  schedule,  and  performance  commitments  upon  which 
their  approval  was  justified).  For  programs  such  as  SPS,  DOD  required  this 
cost,  schedule,  and  performance  information  to  be  reported  quarterly  to 
ensure  that  programs  did  not  deviate  significantly  from  expectations.14  In 
effect,  these  requirements  and  guidance  recognize  that  one  cannot  manage 
what  one  cannot  measure. 


Other  Studies  Reported 
Similar  Findings  and 
Identified  Other  Concerns 


Shortly  after  receiving  our  draft  report  for  comment,  the  department 
initiated  several  studies  to  determine  the  program’s  current  status,  assess 
program  risks,  and  identify  actions  to  improve  the  program.10  These 
studies  focused  on  such  areas  as  program  costs  and  benefits,  planned 
commitments,  requirements  management,  program  office  structure,  and 


An  January  2001,  DOD  issued  a  change  to  its  major  system  acquisition  policy  requiring 
incremental  investment  management.  Specifically,  the  policy  notes  that  a  program’s 
milestone  decision  authority  must  verify  that  each  increment  meets  part  of  the  mission 
need  and  delivers  a  measurable  benefit,  independent  of  future  increments. 
1JClinger-Cohen  Act  of  1996,  Public  Law  104-106,  and  OMB  Circular  A-130  (November  30, 
2000). 

14DOD  Interim  Regulation  5000.2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  and  Major  Automated  Information  System  Acquisition  Programs  (January  4, 
2001). 

15See  for  example,  SPS  Contract  Review:  Preliminary  Report  and  Status,  August  1,  2001; 
The  Present  State  of  the  SPS  Program,  Software  Engineering  Institute,  October  19,  2001; 
and  Independent Reiiew  of the  Standard  Procurement  System  Program ,  Gartner 
Consulting,  November  29,  2001. 
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systems  acceptance  testing.  Consistent  with  our  findings  and 

recommendations,  these  studies  identified  the  need  to 

•  establish  performance  metrics  that  will  enable  the  department  to 
measure  the  program’s  performance  and  tie  these  metrics  to  benefits 
and  customer  satisfaction; 

•  clearly  define  organizational  accountability  for  the  program; 

•  provide  training  for  all  new  software  releases; 

•  standardize  the  underlying  business  processes  and  rules  that  the 
system  is  to  support; 

•  acquire  the  software  source  code;  and 

•  address  open  customer  concerns  to  ensure  user  satisfaction. 

In  addition,  the  department  found  other  program  management  concerns 

not  directly  within  the  scope  of  our  review,  such  as  the  need  to 

•  appropriately  staff  the  program  management  office  with  sufficient 
resources  and  address  the  current  lack  of  technical  expertise  in  areas 
such  as  contracting,  software  engineering,  testing,  and  configuration 
management; 

•  modify  the  existing  contract  to  recognize  that  the  system  does  not 
employ  a  commercial-off-the-shelf  software  product,  but  rather  is 
based  on  customized  software  product; 

•  establish  DOD-controlled  requirements  management  and  acceptance 
testing  processes  and  practices  that  are  rigorous  and  disciplined;  and 

•  assess  the  continued  viability  of  the  existing  contractor. 


DOD  Has  Begun 
Addressing  Problems, 
But  SPS’s  Future 
Remains  Uncertain 


To  address  the  many  weaknesses  in  the  SPS  program,  we  made  several 
recommendations  in  our  July  2001  report. 16  Specifically,  we  recommended 
that  (1)  investment  in  future  releases  or  major  enhancements  to  the 
system  be  made  conditional  on  the  department  first  demonstrating  that 
the  system  is  producing  benefits  that  exceed  costs;  (2)  future  investment 
decisions,  including  those  regarding  operations  and  maintenance,  be 
based  on  complete  and  reliable  economic  justifications;  (3)  any  analysis 
produced  to  justify  further  investment  in  the  program  be  validated  by  the 
Director,  Program  Analysis  and  Evaluation;  (4)  the  Assistant  Secretary  of 


16GAO-01-682,  July  31,  2001. 
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Defense  for  Command,  Control,  Communications,  and  Intelligence  (C3I) 
clarify  organizational  accountability  and  responsibility  for  measuring  SPS 
program  against  commitments  and  to  ensure  that  these  responsibilities  are 
met;  (5)  program  officials  take  the  necessary  actions  to  determine  the 
current  state  of  progress  against  program  commitments;  and  (6)  the 
Assistant  Secretary  of  Defense  for  C3I  report  by  October  31,  2001,  to  the 
Secretary  of  Defense  and  to  DOD’s  relevant  congressional  committees  on 
lessons  learned  from  the  SPS  investment  management  experience, 
including  what  actions  will  be  taken  to  prevent  a  recurrence  of  this 
experience  on  other  system  acquisition  programs. 

DOD’s  reaction  to  our  report  was  mixed.  In  official  comments  on  a  draft  of 
our  report,  the  Deputy  CIO  generally  disagreed  with  our 
recommendations,  noting  that  they  would  delay  development  and 
deployment  of  SPS.  Since  that  time,  however,  the  department  has 
acknowledged  its  SPS  problems  and  begun  taking  steps  to  address  some 
of  them.  In  particular,  it  has  done  the  following. 

•  The  department  has  established  and  communicated  to  applicable  DOD 
organizations  the  program’s  chain-of-command  and  defined  each 
participating  organization’s  responsibilities.  For  example,  the  Joint 
Requirements  Board  was  delegated  the  responsibility  for  working  with 
the  program  users  to  define  and  reach  agreement  on  the  needed 
functionality  for  each  software  release. 

•  The  department  has  restructured  the  program  office  and  assigned 
additional  staff,  including  individuals  with  expertise  in  the  areas  of 
contracting,  software  engineering,  configuration  management,  and 
testing.  However,  according  to  the  current  program  manager, 
additional  critical  resources  are  needed,  such  as  two  computer 
information  technology  specialists  and  three  contracting  experts. 

•  It  has  renegotiated  certain  contract  provisions  to  assume  greater 
responsibility  and  accountability  for  the  requirements  management 
and  testing  activities.  For  example,  DOD,  rather  than  the  contractor,  is 
now  responsible  for  writing  the  test  plans.  However,  additional 
contract  changes  remain  to  be  addressed,  such  as  training,  help-desk 
structure,  facilities  support,  and  system  operations  and  maintenance. 

•  The  department  has  designated  a  user-satisfaction  manager  for  the 
program  and  defined  forums  and  approaches  intended  to  better  engage 
users. 

•  It  has  established  a  new  testing  process,  whereby  program  officials 
now  develop  the  test  plans  and  maintain  control  over  all  software 
testing  performed. 
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In  addition,  SPS  officials  have  stated  their  intention  to 

•  prepare  analyses  for  future  program  activities  beyond  those  already 
under  contract,  such  as  the  acquisition  of  additional  system  releases, 
and  use  these  analyses  in  deciding  whether  to  continue  to  deploy  SPS 
or  pursue  another  alternative; 

•  define  system  performance  metrics  and  use  these  metrics  to  assess  the 
extent  to  which  benefits  have  been  realized  from  already  deployed 
system  releases;  and 

•  report  on  lessons  learned  from  its  SPS  experience  to  the  Secretary  of 
Defense  and  relevant  congressional  committees. 

The  department’s  actions  and  intentions  are  positive  steps  and  consistent 
with  our  recommendations.  However,  much  remains  to  be  accomplished. 
In  particular,  the  department  has  yet  to  implement  our  recommendations 
aimed  at  ensuring  that  (1)  future  releases  or  major  enhancements  to  the 
system  be  made  conditional  on  first  demonstrating  that  the  system  is 
producing  benefits  that  exceed  costs  and  (2)  future  investment  decisions, 
including  those  regarding  operations  and  maintenance,  be  based  on  a 
complete  and  reliable  economic  justification. 

We  also  remain  concerned  about  the  future  of  SPS  for  several  additional 
reasons.  First,  definitive  plans  for  how  and  when  to  justify  future  system 
releases  or  major  enhancements  to  existing  releases  do  not  yet  exist. 
Second,  SPS  officials  told  us  that  release  4.2,  which  is  currently  under 
contract,  may  be  expanded  to  include  functionality  that  was  envisioned  for 
releases  5.0  and  5.1.  Including  such  additional  functionality  could 
compound  existing  problems  and  increase  program  costs.  Third,  not  all 
defense  components  have  agreed  to  adopt  SPS.  For  example,  the  Air  Force 
has  not  committed  to  deploying  the  software;  the  National  Imagery  and 
Mapping  Agency,  the  Defense  Advanced  Research  Projects  Agency,  and 
the  Defense  Intelligence  Agency  have  not  yet  decided  to  use  SPS;  and  the 
DOD  Education  Agency  has  already  adopted  another  system  because  it 
deemed  SPS  too  expensive. 


In  summary,  effective  investment  in  information  technology  depends  on 
organizations  (1)  justifying  programs  via  incremental  business  cases  that 
are  based  on  reliable  data  and  sound  analysis,  (2)  making  decisions  on 
investments  in  programs  on  an  incremental  basis,  and  (3)  monitoring 
actual  return  on  investment  (benefits  achieved  and  costs  incurred)  for 
each  increment  and  using  this  information  to  facilitate  decisionmaking 
about  future  increments.  In  the  case  of  SPS,  this  has  not  occurred.  While 
DOD  has  begun  taking  steps  to  strengthen  its  management  of  certain 
aspects  of  the  program  and  committed  to  strengthening  its  investment 
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Contact  and 
Acknowledgement 


management  practices,  questions  still  remain  as  to  what  will  be  done  and 
when.  To  increase  the  chances  of  program  success,  the  department  must 
expeditiously  follow  through  on  its  stated  commitments  and  address  each 
of  our  recommendations.  If  it  does  not,  it  risks  acquiring  and  deploying  a 
procurement  system  that  will  not  produce  business  value  commensurate 
with  costs. 

This  concludes  my  statement.  I  would  be  pleased  to  answer  any  questions 
you  or  Members  of  the  Subcommittee  may  have  at  this  time. 


For  further  information  regarding  this  testimony,  please  contact  Randolph 
C.  Hite,  Director,  Information  Technology  Systems  Issues,  at  (202)  512- 
3439,  or  Cynthia  Jackson,  Assistant  Director,  Information  Technology 
Systems  Issues,  at  (202)  512-5086.  You  may  also  contact  them  by  e-mail  at 
hiter@gao.gov  or  jacksonc@gao.gov,  respectively. 


(310221) 
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